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Introduction 

The  place  to  start  any  meaningful  discussion  of  systems,  as  applied  to  the 
collection,  organization,  and  dissemination  of  information,  or  sources  of  infoma-^ 
tion,  lo  by  defining  them.  The  concept  of  the  information  system  is  one  which  has 
been  very  much  maligned  and  misunderstood.  This  is  due,  in  large  measure,  to  the 
fact  that  we  have  delegated  much  of  our  thinking,  not  to  persons  actively  Involved 
in  the  operation  and  management  of  library  and  information  programs,  but  to  com** 
puter  and  systems  specialists,  who,  all  too  frequently,  try  to  mold  or  contort  in** 
formation  program  requirements  to  make  them  conform  to  the  capabilities  of  tneir 
hardware  or  to  previously  designed  systems,  and  who,  all  too  frequently,  obfuscate 
the  meaning  and  significance  of  the  concept,  as  it  applies  to  libraries  and  infor-* 
station  services. 

We,  in  our  firm,  had  a  very  Interesting  opportunity  in  the  past  year  to  de¬ 
sign  and  operate  a  small-scale  Selective  Dissemination  of  Information  (SDI)  system, 
starting  from  scratch.  1  will  not  go  into  the  details  of  the  design  and  operation, 
but  X  do  want  to  mention  two  things  about  the  system:  First,  the  design,  planning, 
and  debugging  that  went  into  it  consumed  about  two  months  of  a  senior  information 
specialist’s  time,  and  one  month  of  a  computer  programmer's  time.  Second,  we  sent 
out  the  first  notiflcstlonsto  our  sudiencs  within  three  months  after  the  onset  of 
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the  project.  From  the  regular  feedback  wo  are  receiving  from  the  ou<iionce»  the 
system  is  working  very  amoothly;  its  error  ratc>-or  spurious  document  ratc-^is 
about  26  percent,  which  is  s  very  typical  average,  and  its  omission  rate— failure 
to  pick  out  pertinent  documents — is  five  percent,  which  is  somewhat  better  Chan 
average. 

The  reason  for  Che  success  and  economy  of  the  system,  and  the  point  of  this 
digression,  is  tliat  it  was  designed  by  persons  *;lth  an  Intimate  knowledge  of  11» 
brarics  and  Indexing  and  retrieval  oystema.  Long  before  Che  syatem  ever  got  close 
Co  a  computer  or  computer  programmer,  what  it  waa  supposed  to  do  and  how  it  was 
supposed  Co  do  it  had  been  carefully  thought  out  and  documented  by  people  with  a 
firet-hand  knowledge  of  the  processes  involved. 

One  major  weakness  in  whst  we  did  was  Chat  we  were  forced  to  prejudge  the 
need  for  an  SDI  system  sC  Che  onset.  We  we^e  told  Chat  Che  system  had  to  include 
SOI.  We  are  Crying  to  make  up  for  this  by  doing  a  careful  evaluation  of  the  im*' 
pact  of  the  system  on  its  audience,  and  by  comparing  its  utllitjr  against  an  ab** 
sCracClng  and  Indexing  publication  which  we  are  also  producing  for  the  same  olienn. 
We  are  very  fortunate  that  the  design  and  implementation  of  the  SDI  system  cost 
relatively  little.  But  one  is  not  always  that  fortunate.  As  a  rule,  it  is  a  good 
idea  to  avoid  Che  common  trap  of  building  first  and  analysing  afterward.  This  can 
be  very  dangerous  and  costly. 

Systems  Defined 

To  get  back  to  the  subject  of  definitions,  a  system  is  an  ordered,  consistent, 
predictable  way  of  accomplishing  a  desired  end.  The  desired  end  may  be  Che  prepare^* 
Cion  of  food,  Cha  treatmant  of  disasac,  or  tha  ratriaval  or  disaaainacion  of  infor~ 
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Order  and  consisccncy  in  ayatema  give  riao  Co  prcdiccabljllcy,  which,  in 
turn,  gives  rise  co  dcpcndabilicy  and  efficient  use.  Take,  for  Instance,  the 
aubjecC  index  to  Chemical  Abstract*;,  A  user  nay  not  necessarily  ogrec  with  the 
way  a  given  compound  or  phenomenon  is  indexed  in  Chemical  Abstracts .  but  he  gener* 
ally  knows  how  CA  does  it  and  where  to  find  it  In  the  index,  because  CA  generally 
cries  to  do  it  the  same  way.  That  makes  it  a  system. 

Or  Cake  the  case  of  the  classification  systems  used  to  arrange  books  in 
libraries.  One  may  not  agree  with  the  way  books  are  categorised  in  libraries*" 

Che  Library  of  Congress,  for  years,  classified  compucors  under  Calculating  Machines. 
completely  ignoring  non-numerlcal  applications.  However,  you  could  always  depend 
on  books  on  computers  being  shelved  with  books  on  calculating  machines  in  libraries 
using  the  LC  classification  and  this  makes  it  a  system.  It  Is  dependable'—or  per** 
haps  consistently  undependable  would  be  s  better  way  of  putting  it. 

When  I  was  an  undergraduate  caking  a  course  in  Quancitacive  Analysis,  one  of 
Che  firsc  things  we  were  taught  is  Che  difference  between  accuracy  and  precision. 

We  were  cold  that  accuracy  is  being  right  in  the  absolute  sense,  and  precision  is 
being  consistent  in  our  results.  We  soon  learned  that  absolute  accuracy  is  very 
difficult  to  attain,  because  of  imperfections  in  the  instruments  we  worked  wich 
and  a  variety  of  other  factors,  and  that  we  had  to  make  up  for  this  by  being  precise 
or  consistent  In  our  measurements.  This  condition  is,  if  anything,  amplified  in  the 
case  of  the  analysis  and  organization  of  documents,  where  accuracy  in  the  interpre¬ 
tation  of  significance,  usefulness,  and  meaning  lies  mainly  in  the  eye  of  the  be¬ 
holder.  And  so,'  as  in  the  esse  of  quantitative  chemical  measurements,  we  aim  firsc 
for  precision  or  consistency  and  then  for  accuracy.  This  makes  for  a  system. 

To  recapitulate,  a  system  la  a  device  that  is  consistent  and  compatible  wich 
itself  in  doing  what  it  was  established  to  do.  This  is  crue  whether  the  system  is 
the  index  to  s  book,  a  library  card  catalog,  an  entire  library  or  information  center. 
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or,  iot  tliac  mucccr,  d  tiaclonal  or  IntcrnaClonul  network  of  iit>rarlc«  or  InforsMi* 
tion  ccntcra.  A  uyocom  muoc  be  «o  rceponelvo  ee  poesiblo  to  the  ncocli  of  ite 
users.  However,  It  con  never  be  completely  responsive  to  Che  intorescs,  view¬ 
points,  ond  contexts  of  /'ll  of  ice  users;  If  it  were,  it  would  not  be  a  syscea 
but  an  anarchy.  Ultimately,  the  best  of  systens  is  a  eoaproaise,  and  the  ultiaate 
aim  of  a  system  design  is  to  define  and  iaplemenc  the  most  optimal  con.,  toaise. 

Definition  of  System  Reeulrements 

So  much  for  definitions.  Now  how  do  we  go  about  defining,  designing,  and 
implementing  this  optimal  compromise?  I  thought  that  a  good  way  of  diacussing 
this  would  be  to  recount  a  case  from  our  own  recent  experience,  in  which  we  were 
called  upon  to  develop  a  ratner  complex  aystem  for  an  information  center  dealing 
with  a  broad  range  of  the  social  sciences. 

The  first  thing  we  did  was  to  enter  into  detailed  discussions  with  the  mans* 

gets  of  the  institution  to  determine  why  they  thought  they  needed  a  system,  what 

» 

they  wanted  or  expected  it  to  do,  and  what  specific  problems  they  expected  it  to 
solve.  All  too  often,  the  motivation  behind  the  desire  for  a  system  or  program 
is  not  a  clear-cut  definition  of  need  but  a  desire  to  be  in  the  forefront  of  cur¬ 
rent  fashion.  But  all  too  often  what  is  fashionable  for  one  institution  is  quite 
out  of  context  for  another,  and  all  coo  often  fashions  change,  leaving  us  with 
white  elephants  rather  than  viable  working  aechanisoM  that  do  required  Jobe  in 
Che  beat  and  most  efficient  ways. 

vnien  we  had  defined  what,  in  general,  the  system  was  needed  for,  and  what, 
in  general.  It  was  supposed  to  do  and  for  whom,  and  when  we  had  pinned  down  all 
possible  financial  and  adalnistrstlve  constraints  that  woula  affect  or  limit  what 
we  came  up  with,  we  turned  to  the  next  step  in  the  design  procedure,  which  was  a 
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dcCailoU  Interview  etudy  of  the  user  group4  Tho  purpose  of  this  sturdy  was  noc 
merely  to  stick  pins  in  the  users  to  find  out  what  mode  thorn  tick;  nor  was  It 
sn  effort  t,o  cot  tho  use  re  to  design  ,tho  system  for  ua.  although  one  does  get 
many  useful  hints  from  analysing  sueeesoes  and  failures  in  the  use  of  inforaa~ 
tion  tools  and  techniques.  The  user  study  was  addressed  to  very  specific* 
carefully  developed  points  which  would  ultimately  guide  ^  in  defining  rea). 
neede  and  determining  the  best  available  means  of  meeting  them. 

Ingredients  of  Us.ar  Study 

The  first  point  with  which  we  concerned  ourselves  in  Che  user  study  was 
the  exact  activities  and  work«relat«d  subject  interests  of  the  users.  The  pur* 
poss  of  this  line  of  questioning  was  to  guide  the  collection  of  pertinent  litera* 
ture*  and  to  define  sources  of  expertise  within  the  user  group. 

Tl)e  second  point  that  we  dealt  with  in  our  interviews  was  the  exact  words 
and  phrases  used  to  describe  the  subjects  and  activities  encompassed  by  the  audi¬ 
ence.  The  purpose  of  this  line  of  questioning  was  to  help  us  develop  the  beginning 
basis  for  an  indexing  and  retrieval  vocabulary  which  was  reflective  of  the  actual 
working  language  of  the  user  group. 

The  next  series  of  questions  had  to  do  with  the  existing  means  by  which  the 
users  keep  abreast  of  current  projects*  how  they  get  current  research  results* 
how  they  find  out  what  haa  been  done  in  the  past*  how  they  get  answers  to  ques- 
tiens*  and  how  they  generally  get  the  information  they  need  to  do  their  work.  The 
purpose  of  this  section  of  the  interview*  which  was  handled  by  means  of  general  ques¬ 
tions  and  the  collection  of  specific  cease*  was  to  deteralne  what  the  system  had  to 
do  and  what  it  did  not  have  to  do  baceuee  it  was  already  being  don*  eatisfeccorlly. 
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Wc  gained  further  inalghta  «s  to  what  the  syatem  hod  to  do  via  o  ocifies 
of  questions  in  which  we  tried  to  identify  real  and  actual  problems  encountered 
by  the  users  in  crying  to  obtain  the  different  types  of  informoCion  they  needed. 

Our  aim  in  this  series  of  questions  was  to  pinpoint  chose  areas  that  were  In  th« 
most  urgent  need  of  improvement,  and,  in  so  doing,  to  establish  e  schedule  of 
priorities  for  what  the  system  had  to  do. 

A  fifth  sec  of  questions  was  addressed  to  time  variables  chat  might  affect 
the  system  requirements.  Examples  of  such  variables  are:  frequency  of  use  of 
libraries  and  Information  centers,  frequency  of  requests  for  different  types  of 
services,  needed  speed  of  delivery  of  responses  and  alerting  services,  time 
actually  availabla  and  given  to  gathering  and  usa  of  information  and  information 
aources,  and  active  life  of  litetatura  used. 

Another  seriee  of  queetions  vae  designed  to  identify  libraries,  information 
centers,  institutions,  and  individuals  from  which  the  users  had  obcai..ed  Infonaa* 
cion  in  the  recent  past,  and  thoee  to  which  they  had  given  Information.  Our  pur** 
pose  here  was  twofold:  First,  we  were  trying  to  Identify  information  sources  and 
resources  that  could  supplement  those  within  the  information  center  we  were  de¬ 
signing,  and  perhaps  eliminate  certain  of  the  collecting  and  service  functions 
chat  the  center  might  otherwise  have  to  perform.  Second,  we  were  trying  to  identi¬ 
fy  the  probable  character istlce  end  else  of  the  Institutional  and  individual  clien¬ 
tele  that  would  be  served  by  the  center. 

Finally,  as  a  further  meana  of  identifying  existing  sources  and  resources, 
we  asked  a  series  of  quesclone  regarding  the  use  of  group  and  individual  files  and 
collectiona,  again  with  a  view  coward  supplementing  the  facilities  of  the  informa¬ 
tion  ceocsr  and  sllmlaatlng  Jobs  for  it  where  fssslbls. 
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Analvaitt  of  ttackg round  Information 

Following  comploeion  of  our  usor-intcrvlowa—'andt  Incldontally,  since  we 
ar«  talking  about  an  optimal  compromise  and  not  perfection,  such  interviews  need 
not  be  with  all  the  users  but  with  a  representative  cross  section  of  chem—we  did 
a  careful  analysis  and  synthesis  to  see  what  we  had  gotten,  and  to  turn  these 
finding#  into  a  sec  of  requiremente.  1  want  to  emphasite  once  again,  because  this 
tends  to  be  a  bone  of  contention  in  this  field,  that  developed  the  syatem  re~ 
qulremence,  not  the  usere.  However,  we  based  the  sec  of  requirements  that  we 
developed  primarily  on  information  obtained  from  the  usere. 

What  we  found  in  the  way  of  requirements  was  this: 

1.  Tlis  system  had  to  have  the  capability  of  putting  people  in  touch 
with  one  another  directly,  not  via  the  literature,  for  purpose# 
of  consultation  and  quick  access  to  specialized  Information. 

2.  The  system  had  to  provide  information  about  ongoing  research  pro* 
jeecs—in  advance  of  publication  of  their  results. 

3.  The  system  had  to  provide  information  on  current  research  results. 

4.  The  system  had  to  provide  information  on  sources  of  information. 

3,  The  system  had  to  provide  rapid  access  to  literature  that  it  did 

not  own  or  contain. 

6.  The  aystem  had  to  have  a  capability  to  perform  retrospective  searches. 

In  regard  to  searches,  we  found  the  need  for  search  capabilities  in  this 
•Ituatlon  to  be  quits  secondary  to  cspsbillcies  for  current  awareness.  The  reasons 
for  this  were,  first,  because  of  the  dynamism  of  the  subject  or  subjects  we  were 
deallug  with,  and  ascend,  becauss  of  chs  nature  and  habits  of  the  people  who  would 

ba  using  tha  system.  Ths  .potaatlel  ussra  wara,  as  a  rule,  expert#  in  cha  field# 


cUoy  were  dealing  with;  chny  knew  cho  liccracurc  ond  the  i^eoplo  In  their  field* 
iucimatoly;  and  they  were  perennial  •cholars.  Only  when  they  had  tc  look  into 
new  or  alien  fields,  for  one  reaaon  or  other,  did  they  require  retrospective 
scarchos,  and  this  wua  relatively  rare.  More  often,  in  such  instances,  they 
sought  out  specialists  in  the  fields  involved. 

What  we  ended  up  v'lch  was  a  set  of  requirements  which  were  quite  atypical, 
when  viewed  in  terms  of  the  current  mores  of  information  systems.  Tl^e  emphasis 
was  on  interpersonal  communication  and  current  avaroness,  rather  than  on  informa¬ 
tion  retrieval  as  such.  It  is  true  that  past  Is  prologue,  out  in  this  specimen 
field  that  we* re  discussing— Count erinaurgcney-*and  in  many  other  fields,  when  we 
refer  to  the  past  we  mean  the  very  recent  paec,  end  it  appeata  to  be  becoming  ever 
more  recent. 

However,  the  main  point  is  that  we  developed  our  requirements  not  on  the 
besie  of  what  infometlon  systems  ordinarily  do,  but  on  the  basis  of  what  wea 
needed  to  serve  the  identified  needs  of  s  specific  group. 

Selection  of  Methods  or  Mechanisms 

Having  defined  what  the  system  was  to  do,  and  having  checked  it  out  care¬ 
fully  and  in  detail  with  the  client  organisation,  we  turned  to  the  subject  of  how 
to  do  it. 

We  approached  the  subject  of  how  to  make  the  system  go  by  matching  alterna¬ 
tive  means  against  each  of  the  Jobs  that  had  to  be  done.  We  evaluated  each  of 
these  alternative  means  in  terms  of  its  relativa  efficacy  in  getting  the  Job  done, 
the  probability  of  its  accaptanca  by  the  usere,  and  its  cost.  The  importance  of 
the  ability  of  a  given  tool  or  technique  to  perform  its  tasks  efficiently  ie  obvious 
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tho  rolucionoUlp  o£  tUo  cool  or  Coch»i<)uo  Co  ito  uoor&^^iho  pooplo  who  ore  going 
to  bo  uCfoccod  by  ic— and  cho  rolacionohlp  of  tho  cool  or  cochnlquo  co  lea  cruo 
coaCtf  uro  noc  alwuya  oo  obviouo* 

All  coo  ofton.  In  designing  ayscomo  and  8ub>«aysCcm»,  wo  ignore  chc  habit«» 
preferences,  and  idlooyiKraclos  of  cho  people  who  arr  to  ueo  chem,  or  on  whoce 
behalf  we  intend  to  operaCo  chom.  Wo  eeC  up  elcuacione  in  which  we  have  to  change 
the  work  habits  of  the  users  if  chey  ore  co  use  the  eyscen.  As  e  result,  we  fro* 
quently  see  syscems  or  services  chat  are  not  used  enough  to  justify  their  costs. 

It  is  far  more  realistic  co  cry  to  tailor  cho  syscen  co  cho  user  then  the  other 
Way  around. 

In  regard  to  costs,  tho  ulcluace  end  obvious  aim  Is  to  got  the  most  for  the 
snallesc  amount  of  money,  however,  chls  can  be  a  very  elusive  concept.  At  the 
last  ADI  meocing  in  Santa  Monica,  I  participated  in  a  discussion  on  input  mochan- 
isms  for  coaputer'-drivon  composition  devices.  At  a  certain  point  in  the  discussion, 
tho  subject  of  costs  came  up.  One  person  said  that  ho  had  a  device  that  cost  a 
certain  amount  of  money,  could  be  keyed  at  a  certain  race  of  speed,  and  could  be 
Operated  by  very  loW'-lev«l  personnel.  Another  person  got  up  and  said  chat  he  had 
4  device  chat  could  do  everything  that  the  first  one  could  do,  but  it  was  $5,000 
cheaper.  This  was  all  very  illuminating,  but  it  missed  the  most  fundamental  point 
about  costa.  If  a  machine  costa,  say,  $5,000,  and  it  is  used  to  compose  ten  pages 
a  year,  and  it  has  a  working  lifetime  of  five  years,  this  means  that  the  cosfper- 
page,  in  cettns  of  machine  coscn  alone,  ie  $100.  Ti^is  is  obviously  high.  Or.  the 
other  hand,  if  we  had  a  machine  that  cost  $500,000, with  the  some  working  lifetime, 
but  w«  used  it  to  compose  50,000  pages  a  year,  the  cost  per  page  is  $2,  which  is  con- 
aldarably  less  than  $100,  Xn  short,  when  we  determine  cost#,  we  must  think  in  terms 
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•it  t>rlcu  i>oi‘  UMiu  ot  ouC|>ut;,  and  noc  merely  how  much  money  wu  puc  into  the  <;yetom> 
This  Co  SDX  Horvicos,  auotchlng  sorvico*^  and  all  other  funotlons  and  sor* 

Vicos  tliuc  might  be  porfomod  by  the  ayatom. 

One  other  thing  that  wo  should  Chink  about  when  we  eoloct  o  method  or  mechanism 
for  getting  a  Job  done  is  whac,  in  addltlou  to  the  specific  Job,  it  will  contribute 
to  a  syscom.  For  instance,  we  might  find  chat  we  do  not  have  searches  of  sufficient 
logical  complexity  or  in  sufficient  quantity  to  warrant  the  use  of  a  comn.acet  for 
this  purpose^  However,  we  might  still  decide  to  use  a  computer  because  this 
permits  us  to  prepare  accession  lists  and  printed  indexes  rapidly,  it  pennies  us 
to  produce  catalog  cards  cheaply  and  quickly,  and  it  furnishes  us  a  basis  for 
producing  an  SDZ  service,  which  may  be  a  system  requirement.  And  so  wo  always 
look  for  by-products  to  help  us  Justify  or  amortise  whatever  device  or  mechanism 
we  select  to  do  a  Job.  Thia  la  another  way  of  keeping  the  costs  of  our  units  of 
output  as  low  as  possible. 

Design  of  System 

Having  assembled  end  defined  our  processes  and  procedures,  we  turn  to  the 
Job  of  putting  them  together  to  make  them  Into  a  viable  system  that  is,  as  1  men* 
cloned  previously,  compatible  with  itself,  consistent,  and  dependable. 

We  start  this  by  laying  out,  in  detailed  flow  chart  form,  each  of  the  pro* 
cesses  that  arc  to  be  performed  by  the  system.  We  examine  each  flow  chart  for 
each  process  to  make  sure  that  it  is  realistic  and  complece**co  make  sure  that  the 
work  flows  in  logical  and  even  sequence,  that  steps  chat  are  precursors  for  later 
steps  are  treated  as  such,  and  that  there  is  a  minimvuu  of  queuing  or  delay  as  we  go 
from  step  to  step.  All  coo  often,  incomplete  or  foggy  thinking  results  in  irobal* 
antes  or  delays  in  the  work  flow,  and,  as  a  result,  people,  machines,  or  both  are 
forced  CO  remain  U  o  or  become  overloaded  beceuee  of  delays  in  precursor  steps. 
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TtwV«  can  b«  oKtrcnely  coatly,  and  «ust  be  anticipated  and  eliminated  at  the 
oneet . 

When  ve  have  completed  and  debugged  our  aub»eystcms»  we  ore  ready  to  put 
them  together  to  form  an  integrated,  working  eyeCem.  This  stage  is  really  the 
first  moment  of  truth.  This  is  when  we  test  our  logic  and  determine  whether  our 
sub-systems  are  compatible,  and  whether  they  can  operate  in  concert  to  perform  the 
Jobs  that  have  to  be  done,  or  whether  they  are,  in  fact,  standing  in  one  another's 
way.  Zt  sometimes  happens  that  the  steps  involved  in  one  Job  are  in  conflict  with 
the  steps  involved  in  the  performance  of  another.  Situations  like  this  have  to  be 
identified  and  adjudicated  at  this  stage.  Reasonable  compromises  have  to  be  worked 
out  CO  resolve  conflicts  between  processes,  or  else  chaos  will  reign  end  efficiency 
will  suffer. 

Wlten  we  have  completely  debugged  the  system  plan  and  its  constituent  parts, 
we  are  ready  to  turn  it  over  to  our  programmers,  if  computers  are  involved,  and 
CO  implement  it  on  a  pilot  basis.  I  emphasize  the  word  "pilot"  here  because  no 
matter  how  carefully  end  cleverly  we  develop  and  chart  our  design  we  will  inevit¬ 
ably  find,  St  the  Implemencstlon  stage,  that  we  have  done  something  wrong  or  for¬ 
gotten  something.  We  minimize  our  losses  by  avoiding  full-seals  implemencation 
until  we  have  thoroughly  tested  our  design  and  every  paic  of  it. 

Design  Evaluation 

This  leads  us  to  Che  next  step  in  the  procedure:  evaluating  what  we  have 
done.  This  is  the  second  moment  of  truth.  Unfortunately,  the  concept  of  evalua¬ 
tion  in  this  field  is,  if  possible,  even  more  maligned,  misunderstood,  and  misre¬ 
presented  than  Che  coocepn  of  the  system  itself.  Just  as  there  are  vogues  and 
fashions  in  systems— SDZ,  library  machanizetion,  etc.— there  are  vogues  and  fash¬ 
ions  in  methods  of  evaluation.  Yaars  ago,  tha  faahionabla  thing  in  evaluation  was 
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ch«  «o-’CdllcU  library  survey.  Tltcn  va  got  into  so-called  manogemenc  studies. 

Titett  we  get  into  an  era  of  user-studios.  Now  the  vogue  of  the  Moment  is  rele¬ 
vance  and  recall.  Actually,  ell  of  these  techniques  are  useful  Co  e  limited 
degree;  but  each  Cells  only  part  of  the  story. 

Ttiere  are  various  aspects  and  various  levels  of  evaluation,  and  these  all 
have  to  be  considered  and  implemented  if  enything  useful  is  to  be  deduced  from 

the  evaluative  processes.  Evaluation  is  not  merely  whether  a  library  or  informa¬ 

tion  center  has  the  right  kinds  and  quantity  of  books,  whether  its  expenditures 
for  staff  are  in  correct  proportion  to  book  expenditures  or  whether  there  is 
adequate  space  for  books,  staff,  and  readers;  it  is  not  merely  an  analysis  of 
whether  and  how  the  presumed  audience  uses  a  system  or  service;  it  is  not  merely 
a  question  of  how  fast  or  how  well  it  perforv\s  Che  tasks  it  is  presumed  to  have 

Co  do;  and  it  is  not  merely  a  measure  of  .the  false  drops  and  misses  produced  by  a 

retrieval  system.  Xt  is  much  more  than  this.  Xn  order  to  evaluate  a  system,  we 
have  to  find  answers  to  five  basic  questions: 

1.  What  services  should  the  system  be  performing  on  behalf  of  its  users? 

2.  Is  Che  system  performing  all  Che  services  it  should  be  performing? 

3.  Is  the  system  performing  any  services  that  it  should  not  be  performing? 

A.  Is  It  performing  the  services  it  should  bs  performing  as  efficiently 

and  economically  as  pocsibla? 

5.  Xf  not,  what  are  the  causes,  and  what  can  be  done  to  remedy  the  pro¬ 
blem  or  problems? 

Each  of  these  quastlons  requires  a  different  analytical  technique  for  its 
answer. 

The  answer  to  the  first  question,  relating  the  services  chat  the  system  should 
bs  performing,  is  best  obtained  from  analysis  of  the  results  of  user  studies,  since. 
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ultimately^  the  uyetcm  ehould  coma  «•  cloaa  aa  it  can  to  meeting  user  necdu» 
hAbits,  and  proforenecs. 

The  answer  to  the  aacond  queatlon,  relating  to  whether  the  aysecu  is  per- 
terming  all  the  Joba  it  is  supposed  to  requires  a  careful  match  between  the  sys¬ 
tem's  riission  atatoiaent  and  what  It  is  actually  doing.  Xf  the  mission  statement 
la  incomplete  or  out  of  date.  It  may  be  necessary  to  supplement  it  with  informa¬ 
tion  derived  from  the  uaer  survey  end  enalysis  of  requlremente  to  work  up  the 
answer. 

Essentially  the  same  general  procedure  would  apply  to  the  third  question, 
r«  .  ting  CO  Jobs  that  tha  system  ehould  not  be  doing.  The  answer  to  the  fourth 
question,  relating  Co  the  efficiency  end  economy  of  the  system,  ie  derived  by  a 
variety  of  analytical  or  management  techniques  depending  on  what  aspect  or  as'> 
poets  of  the  system  we  are  dealing  with.  Studies  of  relevance  and  recall,  and 
their,  underlying  causes,  might,  for  instance,  be  the  method  of  choice  if  we  arc 
dealing  with  the  searching  and  retrieval  capabilities  of  the  system.  Buc,  then 
again,  another  method  may  be  indicated,  even  in  a  retrieval  situation. 

As  part  of  our  social  science  system  design  assignment,  we  were  required  to 
evaluate  the  retrieval  effectiveness  of  a  file  which  the  clienc-organization  had 
sec  up  to  perform  searches.  The  file  was  of  the  "collectanea"  type,  in  which  each 
page  of  a  document  was  Indexed  by  an  average  of  fivt.  terois  and  a  Xerox  copy  was 
made  and  filed  for  each  term.  Thie  meant  an  average  of  2,000  pages  filed  for  a 
400-page  book,  and,  with  indexing,  copying,  and  filing,  it  meant  about  $800  per 
book. 

In  fulflllmenc  of  the  evaluation  requirement,  we  ran  a  secies  of  test  searches 
and  analyzed  the  results.  We  found  that  the  retrieval  effaetlvenese  of  the  file  was 
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cx'ccllanc:  It  produced  very  few  spuriou*  references  in  its  search  produces^  and 
it  missed  very  few  pertinent  references*  However*  retrieval  power  was  actually 
a  side  issue  in  this  case*  first*  because  it  would  cost  a  prohibitive  amount  of 
money  to  get  a  significant  number  of  documents  into  the  system;  second*  because 
retrospective  retrieval  was  a  secondary  factor  in  the  infonnation>gatherlng 
activities  of  the  group  Involved;  and*  third*  because  the  file  was  cumbersome  to 
use*  and  avoided  wherever  possible  by  its  potential  users. 

Tlic  answer  to  the  fifth  question*  relating  to  the  causes  and  remedies  for 
identified  problems  or  shortcomings  is  largely  a  matter  of  managerial  analysis^ 
of  systematically  reviewing  the  steps  and  processes  involved*  looking  for  weak* 
nesses  or  errors*  and  determining  how  to  correct  them. 

I  would  like  to  say  one  other  thing  about  evaluation.  If  it  is  done  effee* 
Cively***if  it  is  thought  of  as  a  matter  of  quality  control-~lt  is  a  continuing* 
and  never  a  one<-shot  process.  In  a  dynamic  aituation— 'and*  aa  I  mentioned  before* 
situations  in  this  field  tend  to  be  more  and  more  dynamic  and  truth  never  stands 
still — requirements  change*  methodologies  and  technologies  change;  the  best  way 
of  meeting  a  requirement  now  may  become  comparatively  inefficient  later.  People 
operating  systems  change,  end  machines  and  mechanisms  get  old  or  obsolf^te.  And 
so  we  can  never  afford  to  be  sanguins  about  systems.  We  have  to  incorporate  con¬ 
tinuous  and  rigorous  quality  control  procedures  into  their  operation.  That  la  the 
only  way  we  can  be  sure  we  are  doing  the  job  we  set  out  to  do:  to  meet  the  exist¬ 
ing  information  needs  of  our  audience. 

Conclusions 

The  purpose  of  the  foregoing  remarks  has  been  to  remove  some  of  the  mystery 
and  witchcraft  which  have  surrounded  the  concept  of  infocaation  ayatema  for  far 


IS 


too  long »  and  to  urge  the  return  o£  tho  design  and  operation  of  infortnacion  set* 
vices  to  where  they  logically  belong:  to  the  people  who  have  the  dayoto^day  job 
of  making  them  go  and  perforn.  X  have  learned  from  experience  duplicated  over 
and  over  again  cliat  it  does  not  take  a  special  person  with  special  training  to 
produce  an  effoctiva  system;  what  it  takes  is  a  willingness  to  analyse,  carefully 
and  rigorously,  wlmt  is  needed  and  how  to  accomplish  it.  Block  diagrams  and  flow 
charts  are  not  systems;  they  are  mersly  abstract  representations  of  systems.  They 
are  products  of  disciplined  thinking  by  huMn  beings,  and  there  is  einply  no  sub¬ 


stitute  for  thie. 
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«l  ih*  runliMlnti  •iibtofiunrlot,  •i>*fil**,  l>r|i*f|i«irnl  ol  !>*• 
f*A**  *Hl*llv  M  <ilh*»  «fn«nltNll<in  /riiffiiiMl*  iMiilug 

lb*  triMin. 

7b.  nrpoKt  ni;ronry  CLAfisinrA  rioHi  Ktii^Mii*  «**«• 
•H  «*i-u>tl|r  vl«ii*lfli  aHnn  «<  lb*  •*p*M<  lll<ll*Hl*  iwli*lli<‘f 
“K**tt)rl*<|  hill*"  I*  l•l■‘lllll*<l  MnikliHI  I*  !•  Ii*  In  ***«l<b 
»Brr  wMli  *|if«iipfl«l*  **iMiril|i  f*«iil(Hlitfl*< 

]b.  OMOOri  Aii)(i«n«ll<-  dnwfiKriiillHK  I*  •p<icll|*«l  In  Dull  IM* 
l*cllv*  $M(lt  Ml  «nil  Aimrd  F<i><  i>«  IihIu«iiI*(  Mnnunl.  I.iili'r 
(h«  (iiiiup  nMml<*i.  Al««,  wli*n  ii|iplk«hl*,  nhnw  lli*l  <iptliin*l 
nitiklnc*  hcN-i*  liii*n  imH  l«il  Ctiiiiii  ,1  Hnil  tliuup  4  «ulhi«< 
lr*d 

J.  MI'X'OflT  TITUii  Kn**f  lb*  cbmpl*!*  »*p»»l  llll*  In  *11 
r*|<lul  lfll*r*i  TmIi>«  In  *11  cnnr*  •hnttlil  It*  iiiit  lll•Mill<<<l• 

II  •  ni*«nlnt>lul  ml*  ('■mMil  b*  «*le<'l*<l  <«lllinul  vl*«*lllr** 
linn,  «bn»<  llll*  <U**illt«llnn  In  *11  cnpIlnU  In  p  i*«ilb**ln 
In.mi’illaiflp  Inllnwlnn  ||i*  llll*. 

4.  |)l'.5<'|lirTIVK  NCITP^S  It  <i|ipr<iprlii*.  *ni*r  lb*  lyp*  nl 
l*P4rl,  lni*<lni,  pf<it;(***,  «iinifn«rp,  *nnu«t.  nr  lln*l. 

Ulv*  III*  Intiluaiv*  ilnl**  wh*n  N  «pn*lflr  r*|i«rllnn  |l•■tlull  In 
rovrr*i|, 

fl>  AUTIIONIilli  Knt*i  ilir  namrlcl  nl  *ulhi>f<n)  r*  nlmiivn  nn 
tit  In  lh«  r*|i«ri.  p,hl*i  la*!  nani*,  (iral  nam*,  m>i1<li9  Inlllal* 

II  n  llilatri,  aboru  rank  nod  liram  b  til  arfvK*,  Th*  tram*  of 
lb*  prlnrlival  • 'Ibor  in  an  aliaolul*  ailiilmum  r*i|ulr*m*nb 
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70.  TOrAI.  MUMIIIvR  Ok’  I'ACiKS;  Tba  l>i|*l  pap*  tnuni 
abovlil  lollnw  nnraial  paainailan  ploradur**,  I.  *id*r  lb* 
aviaili*!  of  papa*  tnnfalalna  Inlniiaallni^ 

rii.  NUMIllCR  or  lir.Kk.Ki;M(:i'.a  »•*!  Ib*  Inlal  nuadtar  al 

rainri'ni**  *ll*d  In  lb*  rapnrl, 

Ha.  CONIHACT  OK  OMANI  NIIMIIbH;  II  apprnpilaia,  ani** 
Ih*  apidlrabl*  nurabrr  nl  lb*  r  iiairarl  nr  (rani  iindi'i  adili  h 
III*  irporl  waa  willlaiv 

$t>,  K.',  N  td.  PtlOjr.C'r  NIIMIIIvN.  I'.nlr.*  lb*  a|*|irnprlala 
aillllaiy  dfpailairnl  Idi'iiiiliridlnn,  aiirli  ii*  |iru|*i'l  nHiabar, 
*uld>lo|*rl  ntiaili*i.  cym*iii  Mnaibait,  laak  nuaitiai.  *•*• 

90.  ORIOINAtON’H  HI'I’OIIT  NIIMIIS-N(II):  Kniar  Ihe  offi* 

(1*1  raparl  numlinr  by  aibirh  Iba  dncuaiani  will  b*  Idem  Iliad 
and  coMtalled  by  lha  erlniiiall*i|  idlvlly.  Thin  nnmbar  ainal 
bn  anidn*  In  ibU  repurb 

nn.  OniKH  NI  POMT  NUMIIPNISIi  ii  ib*  rapnil  ha*  baan 
aanlpned  any  ether  r*|irrl  nua'bei*  (Blthfrhy  lA*  •ridinatar 
ay  by  Iba  apanaar),  elaa  aalar  IM*  nuaibatltb 

10.  AVAILAIItl.irir/MMITATION  NOTICKb!  Kniaf  any  Mae 
iiniian*  w>  luribar  4i*k*mlnnllnn  9f  Ih#  l•p•ll|  •1h#»  Hwh  lhii»# 


linpa**il  by  aacnrlly  t laaellleallan,  ualny  alaiidard  alalaaianln 
•IM'h  *•! 

<t)  **l>iallll*d  rnpiaal***  may  abiain  capia*  ol  Ihl* 
liimrl  frnm 

(2)  "I'lirplKn  aiinnbncamaiil  and  diaatmlnallan  al  Ibia 
ri'IHiri  by  Dllt.’  la  itel  aulhnrlaad." 

Ol  *'tl.  A.  tiavainmam  apandaa  m*y  ablain  caplaa  nl 
Ibla  rapiirt  dlrablly  Inrm  OllC.  Olbat  <|w*llliad  t>OC 
near*  aball  raqnaal  ibraupb 


"M.  S  ntllllHry  aprnrla*  may  ablain  *npl**  af  Ibl* 
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ahnil  raquaai  ibiauph 
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IJ.  SPONSOHINO  MH.ITAHV  ACTIVITV:  Kmrr  lb*  name  al 
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ward*  but  will  b*  fnllewad  by  an  Indicallon  el  l*rbnlr*|  can. 
laat.  Tha  naaluiwnaM  *1  llnka,  »ala*,  and  woipbl*  I*  nplUnal.  i 
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